{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Count or frequency encoding\n",
    "\n",
    "Another way to refer to variables that have a multitude of categories, is to call them variables with **high cardinality**.\n",
    "\n",
    "We observed in the previous lecture, that if a categorical variable contains multiple labels, then by re-encoding them using one hot encoding, we will expand the feature space dramatically.\n",
    "\n",
    "One approach that is heavily used in Kaggle competitions, is to replace each label of the categorical variable by the count, this is the amount of times each label appears in the dataset. Or the frequency, this is the percentage of observations within that category. The 2 are equivalent.\n",
    "\n",
    "There is not any rationale behind this transformation, other than its simplicity.\n",
    "\n",
    "### Advantages\n",
    "\n",
    "- Simple\n",
    "- Does not expand the feature space\n",
    "\n",
    "### Disadvantages\n",
    "\n",
    "-  If 2 labels appear the same amount of times in the dataset, that is, contain the same number of observations, they will be merged: may loose valuable information\n",
    "- Adds somewhat arbitrary numbers, and therefore weights to the different labels, that may not be related to their predictive power\n",
    "\n",
    "Follow this thread in Kaggle for more information:\n",
    "https://www.kaggle.com/general/16927\n",
    "\n",
    "Let's see how this works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>y</th>\n",
       "      <th>X1</th>\n",
       "      <th>X2</th>\n",
       "      <th>X3</th>\n",
       "      <th>X4</th>\n",
       "      <th>X5</th>\n",
       "      <th>X6</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>130.81</td>\n",
       "      <td>v</td>\n",
       "      <td>at</td>\n",
       "      <td>a</td>\n",
       "      <td>d</td>\n",
       "      <td>u</td>\n",
       "      <td>j</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>88.53</td>\n",
       "      <td>t</td>\n",
       "      <td>av</td>\n",
       "      <td>e</td>\n",
       "      <td>d</td>\n",
       "      <td>y</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>76.26</td>\n",
       "      <td>w</td>\n",
       "      <td>n</td>\n",
       "      <td>c</td>\n",
       "      <td>d</td>\n",
       "      <td>x</td>\n",
       "      <td>j</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>80.62</td>\n",
       "      <td>t</td>\n",
       "      <td>n</td>\n",
       "      <td>f</td>\n",
       "      <td>d</td>\n",
       "      <td>x</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>78.02</td>\n",
       "      <td>v</td>\n",
       "      <td>n</td>\n",
       "      <td>f</td>\n",
       "      <td>d</td>\n",
       "      <td>h</td>\n",
       "      <td>d</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        y X1  X2 X3 X4 X5 X6\n",
       "0  130.81  v  at  a  d  u  j\n",
       "1   88.53  t  av  e  d  y  l\n",
       "2   76.26  w   n  c  d  x  j\n",
       "3   80.62  t   n  f  d  x  l\n",
       "4   78.02  v   n  f  d  h  d"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# let's open the mercedes benz dataset for demonstration\n",
    "\n",
    "data = pd.read_csv('mercedesbenz.csv', usecols=['X1', 'X2', 'X3', 'X4', 'X5', 'X6', 'y'])\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X1 :  27  labels\n",
      "X2 :  44  labels\n",
      "X3 :  7  labels\n",
      "X4 :  4  labels\n",
      "X5 :  29  labels\n",
      "X6 :  12  labels\n"
     ]
    }
   ],
   "source": [
    "# let's have a look at how many labels\n",
    "\n",
    "for col in data.columns[1:]:\n",
    "    print(col, ': ', len(data[col].unique()), ' labels')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Important\n",
    "\n",
    "When doing count transformation of categorical variables, it is important to calculate the count (or frequency = count/total observations) **over the training set**, and then use those numbers to replace the labels in the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((2946, 6), (1263, 6))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(data[['X1', 'X2', 'X3', 'X4', 'X5', 'X6']], data.y,\n",
    "                                                    test_size=0.3,\n",
    "                                                    random_state=0)\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a': 34,\n",
       " 'aa': 1,\n",
       " 'ac': 10,\n",
       " 'ae': 342,\n",
       " 'af': 1,\n",
       " 'ag': 15,\n",
       " 'ah': 3,\n",
       " 'ai': 289,\n",
       " 'ak': 188,\n",
       " 'al': 3,\n",
       " 'am': 1,\n",
       " 'an': 3,\n",
       " 'ao': 10,\n",
       " 'ap': 5,\n",
       " 'aq': 46,\n",
       " 'as': 1155,\n",
       " 'at': 5,\n",
       " 'au': 3,\n",
       " 'av': 2,\n",
       " 'aw': 2,\n",
       " 'ay': 40,\n",
       " 'b': 12,\n",
       " 'c': 1,\n",
       " 'd': 12,\n",
       " 'e': 61,\n",
       " 'f': 59,\n",
       " 'g': 10,\n",
       " 'h': 4,\n",
       " 'i': 15,\n",
       " 'k': 16,\n",
       " 'l': 1,\n",
       " 'm': 284,\n",
       " 'n': 97,\n",
       " 'o': 1,\n",
       " 'p': 1,\n",
       " 'q': 3,\n",
       " 'r': 101,\n",
       " 's': 63,\n",
       " 't': 17,\n",
       " 'x': 8,\n",
       " 'y': 8,\n",
       " 'z': 14}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's obtain the counts for each one of the labels in variable X2\n",
    "# let's capture this in a dictionary that we can use to re-map the labels\n",
    "\n",
    "X_train.X2.value_counts().to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X1</th>\n",
       "      <th>X2</th>\n",
       "      <th>X3</th>\n",
       "      <th>X4</th>\n",
       "      <th>X5</th>\n",
       "      <th>X6</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3059</th>\n",
       "      <td>aa</td>\n",
       "      <td>ai</td>\n",
       "      <td>c</td>\n",
       "      <td>d</td>\n",
       "      <td>q</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3014</th>\n",
       "      <td>b</td>\n",
       "      <td>m</td>\n",
       "      <td>c</td>\n",
       "      <td>d</td>\n",
       "      <td>q</td>\n",
       "      <td>i</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3368</th>\n",
       "      <td>o</td>\n",
       "      <td>f</td>\n",
       "      <td>f</td>\n",
       "      <td>d</td>\n",
       "      <td>s</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2772</th>\n",
       "      <td>aa</td>\n",
       "      <td>as</td>\n",
       "      <td>d</td>\n",
       "      <td>d</td>\n",
       "      <td>p</td>\n",
       "      <td>j</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3383</th>\n",
       "      <td>v</td>\n",
       "      <td>e</td>\n",
       "      <td>c</td>\n",
       "      <td>d</td>\n",
       "      <td>s</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      X1  X2 X3 X4 X5 X6\n",
       "3059  aa  ai  c  d  q  g\n",
       "3014   b   m  c  d  q  i\n",
       "3368   o   f  f  d  s  l\n",
       "2772  aa  as  d  d  p  j\n",
       "3383   v   e  c  d  s  g"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lets look at X_train so we can compare then the variable re-coding\n",
    "\n",
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X1</th>\n",
       "      <th>X2</th>\n",
       "      <th>X3</th>\n",
       "      <th>X4</th>\n",
       "      <th>X5</th>\n",
       "      <th>X6</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3059</th>\n",
       "      <td>aa</td>\n",
       "      <td>289</td>\n",
       "      <td>c</td>\n",
       "      <td>d</td>\n",
       "      <td>q</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3014</th>\n",
       "      <td>b</td>\n",
       "      <td>284</td>\n",
       "      <td>c</td>\n",
       "      <td>d</td>\n",
       "      <td>q</td>\n",
       "      <td>i</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3368</th>\n",
       "      <td>o</td>\n",
       "      <td>59</td>\n",
       "      <td>f</td>\n",
       "      <td>d</td>\n",
       "      <td>s</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2772</th>\n",
       "      <td>aa</td>\n",
       "      <td>1155</td>\n",
       "      <td>d</td>\n",
       "      <td>d</td>\n",
       "      <td>p</td>\n",
       "      <td>j</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3383</th>\n",
       "      <td>v</td>\n",
       "      <td>61</td>\n",
       "      <td>c</td>\n",
       "      <td>d</td>\n",
       "      <td>s</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      X1    X2 X3 X4 X5 X6\n",
       "3059  aa   289  c  d  q  g\n",
       "3014   b   284  c  d  q  i\n",
       "3368   o    59  f  d  s  l\n",
       "2772  aa  1155  d  d  p  j\n",
       "3383   v    61  c  d  s  g"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# And now let's replace each label in X2 by its count\n",
    "\n",
    "# first we make a dictionary that maps each label to the counts\n",
    "X_frequency_map = X_train.X2.value_counts().to_dict()\n",
    "\n",
    "# and now we replace X2 labels both in train and test set with the same map\n",
    "X_train.X2 = X_train.X2.map(X_frequency_map)\n",
    "X_test.X2 = X_test.X2.map(X_frequency_map)\n",
    "\n",
    "X_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where in the original dataset, for the observation 1 in the variable 2 before it was 'ai', now it was replaced by the count 289. And so on for the rest of the categories (compare outputs 5 and 6).\n",
    "\n",
    "### Note\n",
    "\n",
    "I want you to keep in mind something important:\n",
    "\n",
    "If a category is present in the test set, that was not present in the train set, this method will generate missing data in the test set. This is why it is extremely important to handle rare categories, as we say in section 6 of this course.\n",
    "\n",
    "Then we can combine rare label replacement plus categorical encoding with counts like this: we may choose to replace the 10 most frequent labels by their count, and then group all the other labels under one label (for example \"Rare\"), and replace \"Rare\" by its count, to account for what I just mentioned.\n",
    "\n",
    "In coming sections I will explain more methods of categorical encoding. I want you to keep in mind that There is no rule of thumb to indicate which method you should use to encode categorical variables. It is mostly up to what makes sense for the data, and it also depends on what you are trying to achieve. In general, for data competitions, we value more model predictive power, whereas in business scenarios we want to capture and understand the information, and generally, we want to transform variables in a way that it makes 'Business sense'. Some of your common sense and a lot of conversation with the people that understand the data well will be required to encode categorical labels.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
